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Notes in Brief 


Assessing High Impact Practices Using NVivo: 
An Automated Approach to Analyzing Student 
Reflections for Program Improvement 


C 1 

V^/ritical reflection allows students to synthesize their learning and deepen 
their understanding of an experience (Ash & Clayton, 2009). A recommended reflection 
method is for students to write essays about their experiences. However, on a large scale, 
such reflection essays become difficult to analyze in a meaningful way. At Roanoke College, 
we have developed a system to automate the qualitative coding process using a software 
analysis tool (i.e., NVivo). This inductive approach allows us to identify patterns in student 
learning that indicate effective and ineffective aspects of applied learning experiences. It 
provides a more nuanced and rich approach to the analysis of student learning outcomes in 
order to ensure the quality of our programs through continuous improvement. 

Ash and Clayton’s (2009) article on high impact educational practices (HIPs) 
describes the benefits of using student reflection essays as assessment tools and gives 
detailed prompts for eliciting reflective writing. Their DEAL model includes three key 
parts: Describe, Examine, and Articulate Learning. The prompt that is given to students 
provides scaffolding for a deep reflection by asking students to describe a key experience 
using only factual information; examine the experience from three perspectives (academic, 
civil engagement, and personal); and, articulate their experiential learning and how it 
might influence future goals. 

At Roanoke, using the DEAL model provided a framework for students to create 
deep reflections rather than write only surface descriptions. The reflections have revealed 
what students learn from high impact practices. However, using critical reflection essays 
to assess an institution-wide program is difficult, given the length and personal nature 
of student essays. Initially, the program faculty attempted to score sample essays using 
the DEAL Model Critical Thinking Rubric (Ash, Clayton, & Moses, 2007) but found the 
rubric did not capture student voice about learning; therefore, they turned to an inductive 
analytical approach, which allowed them to identify themes that emerged organically from 
student essays and captured the nuance of student experiences. 


Identifying themes in student essays is time-consuming and became infeasible 
when Roanoke College’s experiential learning programs grew to include hundreds of 
students. Although we considered returning to survey assessment or analyzing only a 
sample of essays, we decided instead to utilize NVivo software and develop an automated 
approach to qualitative analysis. 
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NVivo is qualitative analysis software that allows researchers to follow a variety of 
qualitative coding strategies and increase their speed with tools for organizing qualitative 
sources and themes. Additionally, NVivo provides many functions for quickly searching 
essays for specific phrases, analyzing data, and creating figures and graphics. While traditional 
applications of NVivo can be employed to save time while analyzing reflection essays, we 
developed a system using the software’s text search query function that completely automates 
the process. We developed our system using 171 research reflection essays written by students. 
Then, given the early success of our automated coding process, additional NVivo search 
queries were developed to analyze the other high-impact practices in the experiential learning 
program at Roanoke College (i.e., internships, service-learning, and study-away). 

Method 


While traditional 
applications of NVivo 
can be employed to save 
time while analyzing 
reflection essays, we 
developed a system 
using the software’s 
text search query 
function that completely 
automates the process. 


Our automated system uses a framework of syntax developed to work with NVivo’s text 
search query function. Before developing syntax, the different research themes were organized, 
and fourteen themes (identified in Table 1) emerged organically from the student essays. When 
identifying and organizing themes, care was taken to be specific, knowing that themes could 
later be combined, so that no data would be missed when automatically searching. 


Table 1 

Research Essay Themes 

Category 

Theme 

Collaboration with others 

Relationship with advisor 

Collaboration with peers 

Collaboration with staff and other experts 

Gained skills/knowledge of field 

Applied coursework 

Graduate school/career preparation 

Improved presentation skills 

Personal growth 

Increased confidence 

Overcame obstacles 


Increased self-efficacy 

Improved time-management 

Other 

Presenting served as a learning experience 
Encountered obstacles and frustrations 

Wants to continue research 

Struggled with time-management 


We have found that when 
our automated system 
is coupled with a person 
who checks the search 
query results, we are able 
to identify all themes 
with complete accuracy. 


While identifying the themes that emerged in the essays, an inventory of the word combinations 
used to describe each theme was organized. This inventory was later developed into a framework 
of syntax. We began developing the framework using a sample of 49 of the 171 student essays, 
which we first coded manually using a grounded theory approach (Strauss & Corbin, 1998). 
Once the essays were manually themed, each theme was analyzed for the syntax students 
used to describe it. The text search query function within NVivo was then used to search for 
the syntax corresponding with each theme until the query function yielded the same results as 
found in manual analysis. A text search query was developed for each corresponding theme. 
An example of a theme and its corresponding syntax is shown in Table 2. 
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The results of the text search queries were compared to the manually identified 
themes. Once we ensured the results of the text search query were accurate for the first 
sample of 49 essays, we ran a new sample of student essays through the queries. At first, 
the queries were inaccurate, frequently missing themes and falsely identifying other themes 
that were not present. The accuracy of each query was improved by following a process of 
manually coding essays, running the essays through the queries, assessing the accuracy of 
the queries compared with the manual coding, and making small changes to the queries to 
increase accuracy. This process was repeated until all 171 research essays were analyzed both 
manually and automatically using the queries. The text search queries became increasingly 
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Table 2 


Example Theme and Syntax 


Theme 

Query 

Relationship with 
advisor 

"Dr. & question"~15 OR "Dr. & thanks"~9 OR "Dr. & 
excellent"~15 OR "Dr. & amazing"~9 OR "working & closely"~3 

OR "one-on-one" OR "worked & closely"~9 OR "discussing & 

Dr."~8 "advisor & explained"~9 OR "Dr. & mentor"~9 OR 
"professor & closely"~5 OR "Dr. & collaboration"-^ OR "Dr. & 
collaborate"~9 OR "Dr. & collaborated"~9 OR "Dr. & 
collaboratively"~6 OR "Dr. & support"~9 OR "Dr. & 
understanding'^ OR "Dr. & helpful"~9 OR "advisor & 
collaboration"-^ OR "collaboratively & Dr."~5 OR "met & 
weekly"~5 OR "learned & mentor"~5 "help & mentor"~5 OR "Dr. 

& meetings"~5 OR "Dr. & always"~5 OR "one & professor"-4 OR 
"faculty & guide"~8 NOT" rather & Dr."~5 OR "support & 
advisor"~9 OR "Dr. & together"~9 OR "Dr. & guidance"~9 


accurate each time an additional set of essays was analyzed, until no themes were missed with 
the automated coding. In order to further improve the accuracy of our system, text search 
queries were skewed so that the searches identify all themes present in the essays, along with 
extra essays where themes may not be present; this allows for the possibility to review query 
results and filter out any wrongly identified themes. We have found that when our automated 
system is coupled with a person who checks the search query results, we are able to identify 
all themes with complete accuracy. A summary of the process for developing each query is 
visually outlined in Figure 1. 


Implications 

The NVivo query approach led to increased efficiency in the assessment of most HIPs 
included in the experiential learning program at Roanoke College. Developing the system 
required an undergraduate student assistant to work full-time on this project for three weeks. 
The automated system is currently being used to effectively analyze undergraduate research, 
internship, and service-learning essays. A student assistant can automatically analyze 
reflection essays, run analyses on the data, and write a report on the findings in approximately 
one to two days. 

Although effective for several HIPs, the methodology did not produce reliable results 
for study-away reflection essays. When analyzing study-away reflections, we followed the same 
approach outlined in Figure 1 but found the results of the queries to be consistently inaccurate. 
We believe this is because students participating in study-away programs at Roanoke College 
were required to reflect, but they were not given a structured prompt (e.g., Ash & Clayton’s 
[2009] DEAL model). This lack of structure led to inconsistent and often vague reflections 
that were difficult to analyze with text queries. Therefore, we recommend that those trying to 
emulate our automated analysis provide their students with structured prompts. 

Initially, we had concerns that in automating the process, the integrity and authenticity 
of the reflection essays would be lost. We recognize that the automated approach does not 
replace a manual and in-depth analysis of student essays for a deep understanding of individual 
student growth; however, from an assessment standpoint, increasing the speed of qualitative 
analysis allows for more responsive program improvements based on student learning data. 
Prior to automated coding, student assistants analyzed qualitative data at the end of each 
semester, and there was not enough time to review the data and implement changes before 
the beginning of the next semester. Additionally, because qualitative data is coded using 
consistent text search queries, more comparisons can be made across programs. Furthermore, 
an efficient approach to qualitative analysis eliminates the need to analyze only a sample of 
essays, a practice that will likely result in missed opportunities for improvement. 

Based on our findings, we believe our automated system is applicable to institutions 
and programs interested in analyzing reflection essays for assessment purposes. However, 


An efficient approach 
to qualitative analysis 
eliminates the need to 
analyze only a sample 
of essays, a practice 
that will likely result in 
missed opportunities for 
improvement. 


Automated qualitative 
analysis holds promise 
for institutions of all 
sizes, even large univer¬ 
sities where qualitative 
assessment practices 
may seem impractical. 


Volume Nine I Summer 2014 


•EPA 99 






RESEARCH & PRACTICE IN ASSESSMENT 


1. Identify themes present in essays 




f \ 

2. Develop syntax that corresponds with each theme 
based on the word combinations used in essays 

V_ ) 


3. Import 
Essays 



Manually 

identify 

themes 



7. Tweak 
syntax 




5. Run 
essays 
through 
syntax 

6. Assess 
accuracy of 
syntax 



Figure 1. Developing a framework of text search queries. Repeat steps 3 through 7 until text 
search queries are completely accurate. 

student reflection essays must be structured by prompts, and researchers must develop a 
unique system of themes and corresponding syntax specific to the experience. Automated 
qualitative analysis holds promise for institutions of all sizes, even large universities 
where qualitative assessment practices may seem impractical. The automated system is 
highly adaptable, with the query method working well for most high impact practices at 
Roanoke College. 
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